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Alternative  Designs  for  a  Joint  Command,  Control,  Communications,  Computers, 
and  Intelligence  (C4I)  Capability  Certification  Management  (JC3M)  System 

Abstract 

US  DoD  has  tended  to  design  Command  &  Control  (C2)  systems  without  consideration 
for  them  to  interoperate  for  synergistic  effects;  each  designed  for  one  warfighting 
function.  As  systems  have  grown  biologically  into  a  System  of  Systems,  achievement  of 
mission-level  effects  has  disappointed.  Architecting  the  C2  SoS  as  a  whole  is 
improbable.  However,  capabilities-based  acquisition  requires  interoperability 
certification  based  on  delivering  a  war-fighter  capability  via  SoS.  Students  at  the  Naval 
Postgraduate  School  examined  this  problem.  Their  result  is  the  “Joint  Capability 
Command  and  Control  Management  (JC3M)”  system.  This  paper  summarizes  their 
efforts.  A  systems  engineering  process  was  applied  to  elicit  requirements,  create  and 
simulate  alternative  solutions,  and  recommend  a  solution  with  life-cycle  cost  estimates. 
The  simulation  tools  selected  to  support  the  project  were  CORE  to  model  function  and 
data  flow;  Arena  for  timing  and  resource  utilization;  and  POW-ER  (Project, 

Organization,  Work  for  Edge  Research)  for  organizational  design  and  processes.  The  use 
of  these  tools  to  complement  each  other  is  unique.  Results  indicated  that  JTEM 
Capability  Test  Methodology  (CTM)  was  projected  to  have  better  performance  than  other 
alternatives,  with  the  median  LCC.  The  final  recommendation  is  to  monitor  JTEM  CTM 
for  further  maturation  as  it  promises  improvements  in  the  utility  of  C4I  SoS  evaluations. 

Keywords:  interoperability  assessment,  modeling,  systems  engineering 

Introduction 

Across  the  US  Department  of  Defense  (DoD),  early  C4I  systems  were  designed, 
acquired,  and  fielded  independently.  Each  addressed  a  single  warfighting  function,  such 
as  logistics,  fire  support,  or  intelligence.  Over  time,  warfighting  has  grown  in 
complexity,  tempo,  and  scope.  Complex  endeavors  are  characterized  by  participants 
from  not  only  different  services  but  from  different  functional  areas.  They  must  respond 
with  agility  across  a  spectrum  of  action  and  across  smeared  boundaries  between  tradition 
levels  of  warfare.  The  current  scenario  requires  a  network-centric  force,  which  in  turn 
requires  true  C2  interoperability. 

Individual  C41  systems,  most  not  designed,  acquired,  or  managed  as  a  collective 
enterprise,  are  being  integrated  as  such  and  forming  an  interdependent  entity,  a  System  of 
Systems  (SoS),  in  which  emergent  behavior  dominates  and  capability  delivery  cuts  across 
system  boundaries.  System-level  acquisition  and  testing  only  result  in  individual  systems 
meeting  specific  performance  requirements.  The  Joint  Interoperability  Test  Command 
(JITC)  tests  for  end-to-end  connections  “in  the  most  operationally  realistic  environment 
possible”  (rather  than  delivery  of  desired  capability)  to  assess  interoperability. 

Successful  information  exchange  results  in  “certification.”  This  is  the  baseline  system 
for  DoD  interoperability  certification.  However,  complex  interactions  of  effects  drive 
changing  configurations  of  C4I  SoS  with  no  formally  established  requirements  for 
performance  evaluation.  Capability-based  testing  of  a  SoS  is  not  well  understood. 
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However,  the  principle  to  ensure  interoperability  through  testing  during  development 
[National  Research  Council]  is  still  valid. 

The  baseline  interoperability  certification  process  is  inadequate  because  it  does  not 
address  how  the  actual  SoS  supports  complex  endeavors.  Recent  revision  to  the  Joint 
Capabilities  Integration  and  Development  System  emphasizes  that  true  interoperability  is 
characterized  by  “end-to-end  operational  effectiveness  ...  for  mission  accomplishment” 
[CJCSI  3170. OIF],  Guidance  for  writing  Capability  Development  Documents  (CDD) 
requires  Net-Ready  Key  Performance  Parameters  that  assess  “the  net-ready  attributes 
required  for  both  the  technical  exchange  of  information  and  the  end-to-end  operational 
effectiveness  of  that  exchange”  [DoDD  4630.05].  This  is  consistent  with  the  NATO 
definition  of  interoperability  [NATO  CoBP]  and  that  proposed  by  the  Software 
Engineering  Institute  [Kasunic  and  Anderson],  Capability  Portfolio  Managers 
[DEPSECDEF]  and  Functional  Capabilities  Boards  [CJCSI  3170. OIF]  play  a  role  in 
capabilities-based  cross-program  interoperability.  Even  so,  no  system  exists  to  assess  the 
capability  of  a  SoS  requiring  integration  of  functions  and  interfaces  across  multiple 
systems.  A  JC3M  system  is  thus  important  because  it  provides  a  process  for  test  planning 
to  verify  true  interoperability.  It  documents  traceability  between  capabilities  and 
construction,  and  provides  confidence  that  the  C4I  SoS  works. 

In  response  to  this  need,  the  Joint  Test  Evaluation  Methodology  (JTEM)  team  is 
addressing  Joint  SoS  interoperability  testing  at  the  Office  of  Secretary  of  Defense  (OSD) 
level.  Marine  Corps  Systems  Command  (MARCORSYSCOM),  the  acquisition 
organization  for  the  Marine  Corps,  is  approaching  the  issue  from  a  Service  perspective. 
MARCORSYSCOM  has  tasked  the  Marine  Corps  Tactical  Systems  Support  Activity 
(MCTSSA)  to  develop  Marine  Air  Ground  Task  Force  (MAGTF)  C4I  Capability 
Certification  Testing  (MC3T),  a  methodology  for  managing  the  MAGTF  C4I  SoS  as  a 
single  system.  MC3M  will  manage  the  MAGTF  C4I  SoS  as  a  set  of  SoS-level 
capabilities,  rather  than  as  a  fixed  hardware  or  software  baseline. 

NPS  students  assigned  to  the  JC3M  project  team  adopted  a  systems  engineering  approach 
to  the  problem  of  architecting  a  C4I  SoS  assessment  system  that  will  identify  desired 
effects-based  capabilities  and  ensure  the  system  under  test  meets  those  requirements. 

The  JC3M  project  sought  a  life-cycle  balanced  solution  for  existing  test  organizations. 
The  processes  can  be  utilized  by  Service  and  Joint  test  agencies. 


Approach  Description 

The  student  design  team  adapted  several  different  systems  engineering  process  models 
[Acosta,  et  al]  and  tailored  them  to  this  problem.  As  illustrated  in  Figure  1,  it  begins  with 
identifying  a  customer’s  needs  and  proceeds  through  several  phases  until  a  final  solution 
is  recommended.  One  can  see  this  is  a  modification  of  INCOSE’s  SIMILAR  (State  the 
problem,  Investigate  alternatives,  Model  the  system,  Integrate,  Launch  the  system,  Assess 
performance,  and  Re-evaluate)  process  model  [INCOSE]  that  incorporates  elements  of 
the  Systems  Engineering  and  Design  Process  [Paulo]  taught  at  USMA  and  at  NPS. 
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Figure  1:  A  tailored  systems  engineering  process 

During  the  problem  refinement  phase,  research  into  the  problem  space  was  conducted, 
stakeholders  were  identified  and  interviewed,  functional  decomposition  was  started,  and  a 
value  system  was  developed.  Based  on  the  preliminary  functional  analysis  and  value 
hierarchy,  several  alternatives  were  created.  Those  alternatives  were  screened,  and 
ultimately  five  different  alternatives  entered  into  modeling  and  simulation.  The  predicted 
performance  values  generated  by  the  models  were  used  to  objectively  analyze  those 
alternatives:  comparing  them  to  each  other  along  with  life  cycle  cost  estimates.  The  use 
of  a  LCCE  as  part  of  the  analysis  of  alternatives  in  this  problem  domain  is  vital.  Those 
testers  and  test  planners  must  be  paid  for;  it  matters  little  if  the  final  system  provides  the 
best  solution  if  that  solution  is  unaffordable.  Finally,  a  solution  was  recommended,  along 
with  caveats.  Both  the  JTEM  project  and  MC3T  project  will  make  use  of  those 
recommendations. 

It  should  be  noted  that  this  team  did  an  excellent  job  in  connecting  those  values  identified 
early  by  stakeholders,  supported  by  a  through  functional  analysis,  and  integrated  into  the 
value  hierarchy  the  values  resulting  from  modeling  and  simulation  that  drove  the  final 
decision  process. 


Problem  Refinement  &  Functional  Analysis 

Developing  a  real  problem,  or  effective  need,  in  this  situation  proved  more  challenging 
than  anticipated.  Stating  the  central  issue  so  that  the  stakeholders  would  receive  some 
utility  from  the  final  solution  proved  rather  slippery.  In  fact,  identifying  the  “right” 
stakeholders  was  a  challenge  itself.  From  the  perspective  of  C4I  system  users,  any 
process  to  certify  a  system  is  interoperable  within  a  SoS  adds  value  when  that 
certification  signifies  the  SoS’  ability  to  support  his  complex  endeavor.  Verifying  that  it 
conforms  to  technical  standards  and  can  exchange  data  is  a  necessary,  but  not  sufficient 
prerequisite.  Whereas,  in  the  acquisition  community,  a  program  manager  manages 
resources  spent  for  certification.  If  test  results  are  compared  to  criteria  outside  the  scope 
of  his  program  or  not  explicitly  stated  in  requirements  documents,  there  is  risk  with  little 
gain.  The  test  community  therefore  finds  itself  in  the  middle:  to  be  the  honest  broker 
representing  users,  but  still  able  to  add  value  to  acquirers.  The  team  focused  on  the  test 
community,  along  with  in-house  testers  inside  the  acquisition  community,  as  primary 
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stakeholders.  The  final  list  included  the  Joint  Interoperability  Test  Command  (JITC), 
Marine  Corps  Operational  Test  and  Evaluation  Activity  (MCOTEA),  Army  Test  and 
Evaluation  Command,  Navy  Operational  Test  and  Evaluation  Force,  Air  Force 
Operational  Test  and  Evaluation  Center,  MCTSSA,  and  the  JTEM  Project  team  under  the 
Director  of  Operational  Test  &  Evaluation.  As  this  team  was  mostly  composed  of 
MCTSSA  employees,  a  major  influence  was  the  new  MC3T  project.  It  was  that  project 
that  provided  an  initial  primitive  need  for  a  “system  that  defines  and  compares  System  of 
Systems  performance  measures  to  warfighter  needs  in  an  objective  and  measurable  way” 
[Finn],  They  needed  a  system  that  defined  threshold  values  for  C4I  SoS  performance  in 
operational  war-fighting  terms  and  then  obtaining  those  performance  measures. 

The  team  examined  the  larger  context  of  the  problem  to  find  the  true  underlying  need. 

The  team  researched  the  most  up  to  date  interoperability  certification  and  the  latest 
direction  within  the  DoD  on  realizing  desired  capabilities.  While  the  existing  directives 
and  instructions  seem  clear  in  identifying  roles  and  responsibilities  in  a  traditional  sense, 
little  light  was  shed  on  the  root  of  the  issue.  All  stakeholders  were  queried  on  how  they 
plan  a  C4I  SoS  assessment,  what  resources  are  used  to  do  so,  how  component  systems 
under  test  are  identified,  how  performance  requirements  are  codified,  how  conflicts  are 
resolved,  and  what  metrics  they  use  to  assess  their  own  performance  [Acosta,  et  al].  The 
written  questions  all  sought  to  reveal  how  they  knew  they  got  it  right  and  what  areas  were 
most  ripe  for  improvement.  The  responses  from  JTEM  and  JITC  were  professional, 
insightful  and  frank. 

A  basic  functional  hierarchy  began  to  evolve  around  the  three  major  functions  of 
planning  a  C4I  SoS  evaluation,  conducting  the  evaluation,  and  reporting  results.  The 
identification  and  definition  of  performance  threshold  values  was  of  primary  concern  and 
all  stakeholders  seemed  to  be  completely  satisfied  with  their  ability  to  execute  and  then 
report  on  an  evaluation  event.  Therefore,  the  problem  scope  was  focused  on  the  planning 
phases.  Further  decomposition  resulted  in  a  draft  functional  model,  shown  in  Figure  2 
[Acosta,  et  al]. 


Figure  2:  Initial  JC3M  functional  decomposition 

This  project  focused  entirely  on  function  1.0,  Plan  a  C4I  SoS  Evaluation.  Further 
functional  evaluation  identified  required  inputs  and  outputs  of  the  system,  process 
activation  and  termination,  and  evaluation  measures  for  each  of  the  lowest  level 
functions. 

Eventually,  several  alternatives  were  to  be  compared  to  each  other  objectively.  The  basis 
of  that  comparison  must  be  how  well  they  achieved  the  functional  and  non- functional 
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requirements.  By  combining  a  complete  functional  hierarchy  with  critical  non-functional 
attributes  and  assigning  evaluation  measures  to  each,  a  value  system  was  created.  This 
classic  systems  engineering  paradigm  closed  out  the  initial  requirements  analysis  work. 

A  part  of  that  value  hierarchy,  with  only  those  critical  evaluation  measures  that  were 
eventually  used  in  the  final  comparison  of  alternatives  is  in  Figure  3  [Acosta,  et  al].  This 


is  a  small  sample  of  the  information  gained  through  this  analysis.  However,  it  is  telling 
because  it  codifies  how  we,  as  designers,  will  know  if  we  “got  it  right.” 


Figure  3:  Part  of  JC3M  value  hierarchy 

The  lighter  colored  boxes  indicate  the  evaluation  measures  that  defined  the  needs  for  a  set 
of  modeling  tools  and  that  would  drive  the  final  analysis.  A  more  complete  definition  for 
those  elements  is  provided  in  Table  1. 
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Percentage  of 
Traceable 
Measures 

Days  to 

Plan  Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

JC3M 

Function 

Define  Measures 
1.3.2 

Planning  Results 
1.4.3 

Planning  Results 
1.4.3 

Input  System 
Flexibility 

4.1 

Input  System 
Flexibility 

4.1 

Definition 

Alternative 
generated 
measures, 
traceable  to 
stakeholder 
requirements, 
divided  by  the 
number  of 

measures 
generated  by  the 
alternative. 

Elapsed  time  (in 
days)  of  planning 
for  C4I  SoS 
evaluation 

Assign  an  overall 
quality  level  to 
the  planning 
documents 
produced. 

Divide  percent 
change  in  labor 
hours  to  conduct 
planning  phase  of 
JC3M  by  the 
percent  change  in 
systems  under  test. 

Divide  percent 
change  in  duration 
to  conduct 
planning  phase  of 
JC3M  by  the 
percent  change  in 
systems  under  test. 

Ratio  level  data, 
from  0  -  100% 

Integer  count  >  0 
days 

Ordinal  -  Low, 
Medium,  High 

Ratio  level  data 
from  0  -  qo 

Ratio  level  data 
from  0  -  oo 

Rationale  and 
Relevance 

Identifies 
objectivity  of 
performance 
measures. 

Predicts  SoS 
evaluations  that 
can  be  conducted 
in  a  year. 

Identifies 
predicted  utility 
of  alternative. 

Predicts  changes  in 
cost  of  SoS 
evaluation  based 
on  size. 

Predicts  changes  in 
duration  of  SoS 
evaluation  based 
on  size. 

Performance 

measures 

traceable  to 
doctrinal 

references  will  be 
perceived  as 
objective, 
increasing  the 
value  of  the 
evaluation. 

Alternatives  that 
permit  multiple 

SoS  evaluations 
generate  data  to 
support  fielding 
decisions  sooner. 

Quality  of  the 
planning 
products  drives 
the  overall  value 
of  the  alternative. 

Can  be  used  to 
determine  most 
effective 
alternative  based 
on  SoS  size. 

Can  be  used  to 
determine  most 
effective 
alternative  based 
on  SoS  size. 

Table  1:  Evaluation  measure  details 


Design  Alternatives 

There  were  already  three  alternatives  existing  or  in  the  final  stages  of  development  in 
response  to  the  problem  at  hand.  Additionally,  the  team  sought  to  architect  two 
additional  systems.  This  would  present  the  stakeholders  a  broad  range  of  possibilities, 
while  keeping  the  effort  required  for  modeling  and  simulation  manageable. 

The  first  of  the  known  alternatives  was  the  Federation  of  Systems  (FEDOS)  system  used 
at  MCTSSA  in  2005.  FEDOS  was  designed  to  assess  the  performance  of  C4I  systems 
when  assembled  into  the  MAGTF  C4I  SoS.  FEDOS  began  at  the  order  of  the  Deputy 
Commander  for  C4I  Integration  and  Interoperability  (C4II)  at  MARCORSYSCOM,  who 
tasked  MCTSSA  to  assess  SoS  and  systems  interoperability.  A  working  group  of 
stakeholders  in  the  system  developer  community  decided  what  systems  would  participate, 
what  requirements  were  to  be  tested,  and  the  schedule  of  events  to  include  test  planning, 
test  conduct,  and  results  reporting. 

Because  the  MAGTF  C4I  SoS  was  not  designed  in  compliance  with  an  architecture,  there 
were  no  overarching  SoS  performance  measures  or  threshold  criteria.  This  lack  of 
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doctrinal  performance  criteria  meant  that  MCTSSA  test  personnel  had  to  engage  in  long 
and  at  times  inconclusive  negotiations  with  stakeholders  to  define  threshold  values  that 
were  used  to  measure  performance,  and  determine  if  components  passed  or  failed  the  test. 
The  MARCORSYSCOM  Product  Groups,  responsible  for  developing,  fielding,  and 
supporting  C4I  systems,  were  not  ordered  to  participate  in  FEDOS,  and  a  passing  grade 
was  not  required  for  a  milestone  decision.  It  was  perceived  as  a  no-win  situation  for 
Product  Groups:  after  a  system  had  successfully  passed  Operational  Tests  by 
demonstrating  compliance  with  system-level  performance  requirements  in  their 
respective  CDD  or  equivalent,  FEDOS  tested  component  systems  in  ways  they  had  not 
been  designed  for,  but  how  they  are  used  in  the  field.  The  acquisition  community’s 
perception  was  that  FEDOS  was  a  risk  with  no  off-setting  benefit.  Despite  this  short¬ 
coming,  FEDOS  was  relatively  successful  as  the  first  USMC  event  specifically  designed 
from  the  beginning  as  a  SoS  evaluation 

Because  FEDOS  is  the  only  alternative  solution  that  has  been  used  by  a  C4I  test 
organization  for  a  true  SoS  event,  it  was  considered  the  “status  quo”  or  baseline  JC3M 
alternative  solution.  As  with  all  good  analyses  of  alternatives,  the  first  option  to  consider 
is  “do  nothing,”  or,  in  this  case,  “do  it  like  FEDOS.” 

The  second  alternative  was  MAGTF  C4I  Capability  Certification  Test  (MC3T)  developed 
at  MCTSSA  as  a  replacement  for  FEDOS.  Other  participants  in  MC3T  development 
include  the  Space  and  Naval  Warfare  Center  (SPAWAR)  Systems  Center  in  Charleston, 
S.C.  and  the  Marine  Corps  Combat  Development  Command  (MCCDC).  More 
importantly,  representatives  of  the  MARCORSYSCOM  Product  Groups  have  actively 
participated.  Product  Group  representatives  defined  a  "Capabilities  Package"  complete 
with  system  requirements,  and  DoD  Architecture  Framework  documents  that  depict  the 
systems  under  their  cognizance.  MCTSSA  analyzed  the  Capabilities  Package  and 
produced  a  Consolidated  Requirements  Assessment  (CRA).  The  CRA  was  an  agreement 
between  the  stakeholders  on  what  needed  to  be  tested,  the  required  resources,  and  the 
Information  Assurance  compliance  requirements.  Once  the  CRA  was  approved, 

MCTSSA  produced  a  Technical  Proposal.  The  Technical  Proposal  defined  the  technical 
solution  the  IPT  proposed  to  meet  the  requirements  in  the  Consolidated  Requirements 
Assessment  (CRA)  in:  staffing,  C4I  systems  architecture  design,  monitoring  network 
architecture  design,  test  cases,  data  capture  and  analysis  plan,  information  assurance  plan, 
and  risk  assessment.  The  Technical  Proposal  is  confirmed,  becoming  the  Technical 
Solution  which  makes  up  nearly  90%  of  the  Test  Plan  and  includes  detailed  test 
procedures  with  reference  documentation.  The  most  promising  aspect  of  MC3T  is  that 
MCCDC  and  MARCORSYSCOM  have  developed  truly  integrated  architecture 
framework  products:  the  operational  activities  doctrinally  defined  in  the  Marine  Corps 
Task  List  are  shown  explicitly  to  be  supported  by  specific  systems  working  together.  The 
idea  that  form  should  follow  function  in  designing  for  network-centric  effects-based 
operations  is  consistent  with  the  latest  direction  for  architectures  [DoDAF  vl.5]. 

The  third  alternative  was  JTEM’s  Capability  Test  Methodology  (CTM).  The  purpose  of 
ITEM  is  to  “develop,  test,  and  evaluate  M&P  (Methods  and  Processes)  for  defining  and 
using  a  distributed  LVC  (Live,  Virtual,  and  Constructive)  joint  test  environment  to 
evaluate  system  performance  and  joint  mission  effectiveness  . . .  focus  on  developing  and 
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enhancing  M&P  for  designing  and  executing  tests  of  SoS  .  .  [JTEM  Rock  Drill  Event 
Final  Report],  Figure  4  is  an  IDEFO  representation  of  the  CTM  process  [Acosta,  et  al]. 


Figure  4:  JTEM  CTM  in  IDEFO 

One  of  the  more  promising  aspects  of  JTEM’s  CTM  is  that  test  characterization  explicitly 
examines  requirements  from  families  of  CDDs  in  the  context  of  missions  based  on  the 
Universal  Joint  Task  List  (UJTL)  [CJCSM  3500. 04C]  and  Combatant  Command  standing 
operations  plans  and  orders.  More  detailed  descriptions  can  be  found  in  JTEM’s  Joint 
Test  and  Evaluation  (JT&E),  Capability  Test  Methodology  (CTM)  Method  and  Process 
(M&P)  Model  Description  [JTEM  CTM  M&P],  The  complexity  of  scenarios  developed 
for  the  LYC  test  environment  reflect  real-world  complex  military  action  involving 
disparate  forces  executing  closely-linked  complicated  tasks,  including  operations  other 
than  war. 

Two  new  alternatives  that  offer  significant  differences  from  the  existing  systems  were 
developed.  The  classic  morphological  box  (Zwicky  process)  was  applied,  guided  by  the 
high  level  functions  identified  earlier  and  used  in  part  to  identify  evaluation  measures. 
Nine  different  alternatives  were  initially  defined.  Through  several  screening  iterations 
and  re-evaluation  against  the  root  problem,  only  two  remained.  These  were  titled 
“Systems  Capabilities  Review”  (SCR  Alternative)  and  “Functional  Capabilities  Board” 
(FCB  Alternative). 

The  Systems  Capabilities  Review  (SCR)  alternative  is  a  combination  of  two  of  the 
original  nine  alternatives.  It  is  composed  of  a  group  of  stakeholders:  C4I  SoS  user 
representatives,  test  agency  representatives,  system  developers  and  program  managers. 
The  test  agency  representative  chairs  the  group,  which  meets  as  required  to  support  a  C4I 
SoS  evaluation,  at  the  Systems  Command  level.  Inputs  to  SCR  include  source  documents 
such  as  Capabilities  Development  Documents  (CDD),  Operational  Requirements 
Documents,  Test  and  Evaluation  Master  Plans  (TEMP),  Concept  of  Operations 
documents,  Joint  Integrating  Concepts,  Joint  Operating  Concepts,  and  system  level 
metrics.  The  SCR  first  reviews  SoS  capabilities  specifications,  and  will  examine  the 
systems  engineering  artifacts  already  created  (such  as  supporting  DoD  Architecture 
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Framework  documents  and  technical  performance  measures)  and  creates  a  list  of  implied 
and  stated  SoS  capabilities.  Next,  the  SCR  reviews  system-level  documents  and  creates  a 
system-level  capabilities  list.  Third,  the  SCR  maps  system-level  capabilities  to  SoS 
evaluation  measures.  The  SCR  identifies  gaps  in  the  evaluation  measure  list  and  creates 
the  balance  of  evaluation  measures  necessary  to  evaluate  the  performance  of  the  C4I  SoS. 
Figure  5  [Acosta,  et  al]  illustrates  how  SCR  performs  the  JC3M  subfunction  1.3.2 
“Define  Measures.” 


SCR  Alternative 


Figure  5:  SCR  Alternative  Sub-functions 

The  Functional  Capabilities  Board  (FCB)  alternative  relies  on  an  existing  group,  the 
JCIDS  C2  Functional  Capabilities  Board,  to  define  the  performance  measures  of  the  SoS. 
The  existing  role  of  FCB  is  to  perform  “. .  .organization,  analysis,  and  prioritization  of 
joint  warfighting  capabilities  within  an  assigned  functional  area”  [CJCSI  3170. OIF], 
Inputs  to  the  FCB  Alternative  include  the  UJTL  and  subsets,  Concept  of  Operations 
(CONOPS)  documentation,  acquisition  program  documentation,  and  system  trouble 
reports.  The  additional  effort  proposed  in  this  alternative  represents  an  increase  in  the 
work  performed  by  the  C2  FCB,  but  is  in  the  same  functional  area,  and  engages  in  the 
similar  tasks.  Unlike  the  SCR,  the  FCB  meets  on  an  ongoing  basis,  rather  than  as 
required  to  support  SoS  evaluations.  The  FCB  will  first  identify  the  configuration  of  the 
SoS  by  determining  the  component  systems.  Next,  the  FCB  will  identify  the  SoS 
capabilities.  SoS  CONOPS  are  reviewed  to  determine  evaluation  measures,  and  finally 
the  FCB  will  generate  the  SoS  evaluation  measure  list  for  use  in  C4I  SoS  evaluations.  As 
the  systems  under  the  cognizance  of  the  Joint  Command  &  Control  Capability  Portfolio 
Manager  are  explicitly  listed  [DEPSECDEF],  their  participation  in  this  alternative  would 
be  required.  The  FCB,  under  JCIDS,  has  a  long-term  mandate,  and  provides  a  short  term 
solution  to  the  lack  of  SoS  performance  measures.  The  relationship  between  the  FCB 
and  C4I  test  organizations,  and  the  list  of  sub  tasks  needed  to  complete  the  Define 
Measures  task,  is  illustrated  in  Figure  6  [Acosta,  et  al].  Because  the  FCB  is  external  to 
the  test  organization,  some  analysis  of  the  performance  measures  generated  by  the  FCB 
will  be  necessary.  Additionally,  it  is  understood  that  a  working  group  within  the  FCB 
itself  would  perform  the  required  analysis. 
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FCB  Alternative 


FCB  Team 
external  to 
the  test 
agency 


1.1  Assesses  issues  to 
support  JROC 
recommendations 


1.2  Assess  DOTMLPF 
Change 

Recommendation  (DCR)s 
using  the  following  criteria 


1.3  Make 

recommendations  to  their 
respective  FCBs 
concerning  proposal 
content  and  suitability  for 
presentation  to  the  JCB 
and  JROC. 


Figure  6:  FCB  Alternative  sub-functions 


Both  of  these  new  alternatives  developed  by  the  JC3M  team  rely  on  supporting  integrated 
architectures  and  CONOPS  documentation  in  addition  to  documentation  normally 
examined  as  part  of  C4I  interoperability  test  preparation.  The  difference  between  these 
alternatives  is  in  the  approach  each  alternative  takes  to  complete  process  1.3.2  “Define 
Measures”  in  the  JC3M  Functional  Hierarchy.  The  SCR  alternative  incorporates  all  tasks 
as  part  of  the  test  planning  process.  The  FCB  Alternative  utilizes  an  external  team  that 
meets  years  round  to  provide  capability  measures  to  the  test  agency. 

Five  different  alternatives  had  now  been  defined  in  some  detail  as  well  as  evaluation 
measures  to  be  used  to  compare  those  alternatives.  It  was  now  a  just  a  matter  to 
determine  actual  values  or  values  obtained  from  simulation  models  for  each  alternative. 


Modeling  &  Results 

Modeling  and  simulation  were  used  extensively  in  this  project.  With  the  exception  of 
FEDOS,  no  other  alternative  under  consideration  existed.  The  only  means  to  gather 
performance  data  in  support  of  decision-making,  short  of  “building”  each  alternative,  was 
through  simulation.  It  was  the  most  cost  effective  means  to  obtain  the  required 
evaluation  measures  in  a  repeatable  and  objective  fashion.  Several  different  modeling 
tools  were  used  to  generate  the  necessary  data.  Figure  7  [Acosta,  et  al]  illustrates  which 
tools  were  used  to  obtain  the  evaluation  measures  which  in  turn  supported  later  cost- 
benefit  analysis. 
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Figure  7:  JC3M  M&S  supporting  evaluation  measures 


Models  of  each  alternative  were  built  based  on  the  functional  architectures  already 
created,  and  elements  unique  to  their  physical  instantiations  were  added.  That  is, 
complete  functional  models  in  IDEFO  were  created  with  Vitech’s  CORE  to  support  the 
simulation  models  built  in  Arena  and  POW-ER.  Within  Arena  and  POW-ER,  those 
attributes  that  differentiated  the  alternatives  from  each  other,  such  as  organizational 
structure,  relationships  with  external  systems,  and  processing  of  certain  inputs,  were 
included.  The  IDEFO  view  of  the  systems  actually  proved  very  insightful  in  terms  of 
explicitly  describing  the  relationship  between  the  functions,  at  all  levels  of  abstractions, 
in  terms  of  their  inputs  and  outputs.  The  models  were  then  executed  by  providing  input 
to  simulate  a  system  under  test  along  with  its  supporting  information.  The  results  of 
several  different  iterations  with  variations  in  the  input  data  sets  were  gathered  and  used  to 
populate  the  table  of  evaluation  measures  with  raw  data.  The  “off-line  evaluation” 
indicated  the  use  of  desk-top  evaluation  by  test  and  development  community 
representatives,  similar  to  the  ITEM  Rock  Drills.  One  could  consider  this  a  kind  of 
human-in-the-loop  simulation;  just  another  kind  of  model  or  prototype  that  has  been  used 
successfully  in  this  problem  domain  [ITEM  Rock  Drill  Event  Final  Report], 


POW-ER  (Project,  Organization,  and  Work  for  Edge  Research)  is  a  project  organization 
modeling  and  simulation  tool  that  integrates  organizational  and  process  views.  POW-ER 
was  developed  via  the  Virtual  Design  Team  (VDT)  computational  modeling  research  at 
Stanford  University.  POW-ER  addresses  organizational  elements  that  impact  the  ability 
to  work  effectively,  including:  policies  and  structures  (culture,  communication, 
decisions,  meetings);  staffing,  hiring,  and  training  needs  for  workforce  plans.  Using 
POW-ER,  the  team  modeled  the  organizational  structure,  the  relationship  between 
individuals  within  those  organizations,  and  individual  task  allocations.  Use  of  CORE  to 
support  functional  analysis  proved  most  helpful  because  it  allowed  the  modelers  to 
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represent  the  same  functional  architecture  in  the  refined  IDEFO  models  as  a  functional 
flow  in  FFBD  format.  That  allowed  the  creation  of  PERT-like  sequencing  of  tasks 
required  when  modeling  work  processes  in  POW-ER.  POW-ER’s  ability  to  predict  and 
analyze  backlogs  proved  to  be  quite  useful  in  the  design  and  troubleshooting  of 
alternative  models,  because  it  allowed  the  team  to  identify  backlogs  in  the  workflow  of 
models.  The  analysis  of  backlogs  in  the  workflow  enabled  the  team  to  identify  the 
optimized  arrangement  of  tasks  and  personnel  for  FCB  and  SCR  since  they  were  created 
for  this  project.  No  such  changes  were  made  to  the  other  alternatives.  Based  on  modeler- 
defined  parameters  like  the  amount  of  effort  required  for  each  task,  the  number  of  full¬ 
time  equivalents  available  with  appropriate  skills  and  number  of  hours  in  a  work- week, 
the  POW-ER  simulation  tool  can  calculate  a  project’s  duration  based  on  simulated 
duration.  Simulated  duration  factors  in  the  “hidden  work”  that  traditional  Critical  Path 
Method  does  not.  The  “hidden  work”  associates  an  amount  of  rework  and  delays  into 
each  task  based  upon  random  variables  described  for  each  task  by  the  modeler.  The 
simulated  duration  provided  the  number  of  days  to  plan  an  evaluation  for  each  alternative 
[Acosta,  et  al]. 

Arena  is  a  commercial  tool  available  from  Rockwell  Automation.  It  provides  a  numerical 
evaluation  of  a  system  by  imitating  the  system’s  operations  or  characteristics  over  time. 
Arena  allowed  the  team  to  conduct  numerical  experiments  in  order  to  predict  the 
behavior  of  an  alternative  given  a  set  of  conditions.  Two  evaluation  measures  required 
assessing  the  changes  in  output  as  a  function  of  the  changes  in  inputs:  Elasticity  of  Labor 
and  Elasticity  of  Duration.  Arena  allowed  the  team  to  run  simulations  on  the  alternative 
models  with  varying  sets  of  inputs.  Those  input  data  sets  represent  the  number  of 
systems  with  their  associated  documentation  that  a  SoS  test  event  would  typically  cover. 
The  baseline  data  set  was  that  group  of  systems  used  during  the  FEDOS  event.  It 
included  over  90  systems  such  as  AFATDS,  EPLRS,  GCCS-J,  SINCGARS  and  TBMCS. 
There  were  fourteen  SoS  capabilities  examined  including  blue  force  common  operational 
picture,  call  for  fire,  common  logistics  and  theater  ballistic  missile  tracking.  Variation  in 
the  input  data  set  was  accomplished  by  changing  the  number  of  individual  systems,  the 
number  of  old  SoS  capabilities  and  the  number  of  new  SoS  capabilities  under  test  for 
each  data  set.  The  same  input  data  set  was  used  for  one  run  of  each  alternative,  enabling 
a  true  head-to-head  comparison.  The  model  in  Arena  was  designed  such  that  the 
subprocess  tasks  would  vary  in  duration  based  on  varying  the  input  systems  under  test. 
Thus,  Arena  displayed  the  output  changes  of  the  entire  alternative  process  that 
corresponded  to  each  of  the  varying  inputs.  The  output  changes  (as  a  percent  of  the 
baseline),  compared  to  percent  change  of  the  input  became  the  values  for  elasticity  of 
duration  and  elasticity  of  labor  [Acosta,  et  al]. 

The  models  were  validated  against  actual  data  from  the  FEDOS  event  of  2005.  Because 
the  original  labor  hour  time  sheets  for  planning  that  event  were  available,  validating  the 
models  was  relatively  simple.  That  is,  the  FEDOS  process  model  was  built  in  CORE, 
which  supported  the  more  elaborate  models  in  POW-ER  and  Arena.  Then,  the  outputs 
were  compared  to  the  appropriate  actual  data  from  FEDOS.  The  number  of  labor  hours 
and  calendar  day  predictions  from  Arena  and  POW-ER  were  within  1%  of  the  actual 
values  [Acosta,  et  al]. 
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Figure  8  [Acosta,  et  al]  summarizes  the  entire  simulation  process,  including  inputs  and 
output  values. 

Input  to  Models  Output 


This  study  represents  the  first  time  these  modeling  tools  were  used  together  to 
complement  each  other.  The  simulations  predicted  key  parameters  of  each  alternative 
design.  Without  such  an  approach,  no  objective  or  repeatable  means  to  compare  the 
alternatives  against  the  requirements  in  those  areas  would  have  been  possible.  Because 
the  results  for  the  FEDOS  models  were  validated  against  known  historical  data,  and  the 
other  models  used  elements  from  it  based  on  a  task  mapping  from  each  alternative  back 
to  the  FEDOS  process,  there  is  a  high  degree  of  confidence  in  the  computer-based 
measures. 

There  were  still  two  evaluation  measures  that  could  not  be  determined  by  computer-based 
simulation:  percent  traceable  measures  and  quality  of  planning  outputs.  The  team  was 
able  to  engage  SMEs  from  several  NAVSEA  and  NAVAIR  field  activities  to  participate 
in  assigning  a  value  for  quality  of  planning  products.  They  were  assembled,  presented 
with  all  five  alternatives,  and  allowed  to  ask  questions  to  ensure  clear  understanding  of 
how  each  process  worked  along  with  built  in  limitations.  Each  SME  responded  to 
specific  questions  about  the  predicted  quality  of  planning  products  coming  out  of  each 
process  with  regard  to  their  effectiveness  in  examining  interoperability  within  a  SoS, 
conformance  to  standards,  and  usability.  The  responses  were  based  on  a  4-point  Likert 
scale  for  each  alternative.  Percent  of  traceable  measures  was  simpler  to  determine,  once 
a  key  assumption  was  accepted.  A  proxy  was  defined  to  be  the  number  of  authoritative 
sources  considered  divided  by  the  total  number  of  authoritative  sources  available.  This 
assumption  is  valid  if  there  is  a  linear  relationship  (as  a  set)  between  the  number  of 
measures  created  and  the  number  of  sources  used  in  creating  those  measures. 

The  final  listing  of  the  raw  scores  is  provided  in  Table  2. 
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Percentage  of 
Traceable 
Measures 

Days  to  Plan 
Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity  of 
Labor 

Elasticity  of 
Duration 

(%) 

(Days) 

(1-4  Likert 
Scale) 

(unit  less) 

(unit  less) 

Ideal  Value 

100% 

Less  is  better 

4  is  Ideal 

Less  is  better 

Less  is  better 

FEDOS 

0 

140 

3.17 

0.87 

0.87 

MC3T 

72 

121 

3.25 

0.78 

0.78 

JTEM  CTM 

92 

73 

3.42 

1.04 

0.83 

SCR 

92 

158 

3.00 

0.98 

0.98 

FCB 

88 

127 

2.75 

0.72 

0.72 

Table  2:  Raw  evaluation  measures 


One  should  note  the  extremely  short  duration  to  plan  an  event  for  the  JTEM  CTM 
process.  This  is  to  be  expected  based  on  that  system’s  reliance  on  SMEs  in  so  many 
different  fields,  minimizing  cross-checking  with  multiple  stakeholders.  On  the  other 
hand,  that  alternative’s  elasticity  of  labor  was  the  worst. 

But,  with  so  many  measures,  how  could  a  single  “best”  alternative  be  found?  The  team 
chose  to  apply  classic  multi-attribute  utility  theory  (MAUT).  While  MAUT  has  its  well- 
documented  limitations,  it  presents  a  means  to  compare  the  alternatives  on  a  single 
weighted  sum  of  utilities  associated  with  each  evaluation  measure.  Raw  scores  are 
converted  to  a  value  or  utility  score,  that  value  is  multiplied  by  its  global  weight,  the 
resulting  weighted  values  are  summed  to  a  single  overall  value.  The  same  SMEs  who 
participated  in  the  process  to  obtain  planning  product  quality  figures  also  participated  in 
the  process  to  determined  value  functions  and  swing  weights.  It  should  be  noted  that  this 
team  used  the  mathematically  rigorous  Wymorian  standard  scoring  functions  for  value 
curves  to  convert  raw  scores  to  utility.  Additionally,  they  were  very  precise  about  their 
application  of  swing  weights  and  rigor  of  the  analytical  hierarchy  process  to  obtain 
weights  [Acosta,  et  al].  So,  the  weaknesses  inherent  in  MAUT  were  minimized  via  these 
tools  and  techniques.  The  final  total  scores  are  shown  in  Table  3. 


Percentage  of 
Traceable 
Measures 

Days  to  Plan 
Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity  of 
Labor 

Elasticity  of 
Duration 

Overall 

Utility 

(0-1) 

FEDOS 

0.00 

0.04 

0.39 

0.06 

0.14 

0.63 

MC3T 

0.02 

0.05 

0.39 

0.07 

0.16 

0.71 

JTEM  CTM 

0.24 

0.06 

0.40 

0.04 

0.15 

0.89 

SCR 

0.24 

0.02 

0.37 

0.05 

0.10 

0.79 

FCB 

0.22 

0.05 

0.34 

0.08 

0.18 

0.87 

Table  3:  Overall  utility  of  the  alternatives 


The  last  step  in  the  process  to  bring  together  all  the  elements  of  a  thorough  analysis  of 
alternatives  was  to  create  a  life  cycle  cost  estimate  (LCCE)  for  each  alternative.  All  costs 
associated  with  development,  implementation,  operations  and  support  through  disposal 
and  transition  were  estimated.  Actual  data  from  the  FEDOS  event,  to-date  actual  costs 
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and  to-completion  estimates  (directly  from  their  respective  project  managers)  for 
development  of  JTEM  CTM  and  for  development  of  MC3T  were  relatively  easy  to 
capture,  once  complete  definitions  for  those  phases  and  cost  breakdown  structures  were 
developed.  Because  the  SCR  and  FCB  Alternatives  were  similar  to  MC3T  in  scope  and 
effort,  development  costs  were  based  on  the  MC3T  numbers.  As  operations  and  support 
for  such  a  system  is  dominated  by  labor  costs,  the  annual  cost  for  each  alternative  was 
based  on  applying  the  prevailing  man-hour  rates  to  the  labor  hour  counts  from  the  POW¬ 
ER  models.  Disposal  and  transition  costs  were  assumed  to  be  the  same  for  each 
alternative  because  those  efforts  are  practically  identical  in  terms  of  level  of  effort  and 
duration.  Table  4  summarizes  the  LCCE  for  each  alternative. 


Life-Cycle  Year 

Total  Cost 

Alternatives 

1 

2 

3 

4.. .9 

10 

($) 

FEDOS 

Development 

0 

0 

0 

0 

0 

0 

Implementation 

1,052,527 

0 

0 

0 

0 

1,052,527 

Operational  &  Maint. 

0 

419,497 

419,497 

419,497 

2,200 

3,908,178 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

1,052,527 

419,497 

419,497 

419,497 

52,200 

5,010,706 

MC3T 

Development 

0 

0 

0 

0 

0 

0 

Implementation 

1,169,414 

0 

0 

0 

0 

1,169,414 

Operational  &  Maint. 

0 

525,537 

525,537 

525,537 

2,200 

4,756,500 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

1,169,414 

525,537 

525,537 

525,537 

52,200 

5,975,913 

JTEM  CTM 

Development 

1,030,000 

2,470,000 

0 

0 

0 

3,500,000 

Implementation 

0 

0 

1,169,414 

0 

0 

1,169,414 

Operational  &  Maint. 

0 

0 

0 

558,535 

2,200 

2,253,410 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

1,030,000 

2,470,000 

1,169,414 

558,535 

52,200 

6,972,824 

FCB 

Development 

1,021,835 

0 

0 

0 

0 

1,021,835 

Implementation 

1,301,282 

0 

0 

0 

0 

1,301,282 

Operational  &  Maint. 

0 

650,223 

650,223 

650,223 

2,200 

5,753,985 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

2,323,117 

650,223 

650,223 

650,223 

52,200 

8,127,101 

SCR 

Development 

952,007 

0 

0 

0 

0 

952,007 

Implementation 

1,169,414 

0 

0 

0 

0 

1,169,414 

Operational  &  Maint. 

0 

624,451 

624,451 

624,451 

2,200 

5,547,811 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

2,121,421 

624,451 

624,451 

624,451 

52,200 

7,719,232 

Table  4:  LCCE  summary 


The  JC3M  team  determined  the  most  expensive  alternative  was  the  FCB  Alternative,  at  a 
cost  of  $8.13  million  over  the  ten-year  projected  lifecycle.  The  team  calculated  the  cost  of 
FCB  as  a  cost  to  DoD,  i.e.  while  the  senior  SMEs  who  generate  the  performance 
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measures  do  not  charge  their  efforts  directly  to  a  C4I  test  organizations,  their  time  and 
effort  is  a  cost  to  DoD.  The  team  determined  that  MC3T  was  estimated  to  cost 
approximately  $960,000  more  than  FEDOS,  which  it  has  replaced.  While  this  is  nearly  a 
20%  difference,  the  increase  can  be  directly  attributed  to  the  increase  in  scope,  duration, 
and  level  of  effort  involved  in  MC3T,  which  anecdotally  supported  the  increased  cost  of 
MC3T  [Acosta,  et  al].  More  importantly  is  that  the  development  cost  for  JTEM-CTM  is 
the  largest  (its  development  is  spread  over  several  years).  However,  the  O&S  costs  are 
the  lowest.  This  is  significant  because  a  test  agency  (or  test  branch  within  a  development 
agency)  deciding  between  these  options  would  incur  only  those  costs  to  implement  this 
option  and  then  could  reap  the  benefit  of  keeping  annual  costs  very  low. 


Recommendations 

A  complete  analysis  of  the  alternatives  based  on  the  preceding  data  was  conducted  to 
determine  the  “best”  alternative.  That  is,  which  one  is  projected  to  provide  the  greatest 
utility  for  the  cost?  Figure  9  [Acosta,  et  al]  summarizes  the  results.  Again,  the  utility  is  a 
weighted  sum  of  several  different  attributes,  all  tied  directly  to  the  overall  goal  of 
ensuring  testing  for  true  interoperability,  a  pre-requisite  for  any  C2  SoS  supporting  a 
disparate  networked  force. 


Life  Cycle  Cost  Estimate  ($MIL) 


Figure  9:  Utility  versus  LCC 

The  JTEM  CTM  process  is  projected  to  perform  slightly  better  than  all  the  other  options 
while  having  a  LCCE  less  than  the  two  others  with  closest  utility  scores.  Days  to  plan  an 
evaluation,  quality  of  planning  products  and  percentage  of  traceable  measures  are  those 
attributes  that  drive  this  performance.  It  should  also  be  noted  that  a  nearly  straight  line 
could  be  drawn  between  FEDOS,  MC3T  and  FCB.  That  leaves  the  SCR  Alternative 
below  the  line  and  JTEM  CTM  above  it.  However,  the  better  way  to  examine  this  figure 
is  to  consider  an  efficient  frontier  of  utility  for  every  cost  value.  A  linear  frontier  is 
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formed  by  a  line  connecting  the  points  for  FEDOS,  MC3T,  and  CTM.  Thus,  the  FCB 
and  SCR  points  are  “below”  that  line  -  less  efficient  and  dominated  by  CTM. 

It  must  be  noted  there  is  some  difference  in  the  confidence  we  have  in  the  performance 
measures.  Because  FEDOS  and  MC3T  were  used  in  actual  full-scale  SoS  test  events, 
their  performance  is  based  on  historical  documentation.  JTEM  CTM’s  performance 
measures  are  based  on  desk-top  simulations  called  “rock  drills”  in  which  test  community 
personnel  exercised  certain  aspects  of  that  system  in  an  artificial  scenario.  Additionally, 
members  of  the  JTEM  team  participated  in  this  study,  validating  nearly  every  aspect  of 
JTEM  CTM  that  was  considered  and  confirming  simulation  output  was  as  expected.  The 
results  from  the  SCR  and  FCB  Alternatives  were  purely  from  the  simulation.  However, 
the  simulation  was  based  on  modifying  parts  of  models  validated  through  FEDOS  data. 

With  regard  to  cost,  similar  logic  can  be  applied.  Those  numbers  from  FEDOS  and 
MC3T  are  based  on  actual  costs.  The  cost  estimates  for  the  other  alternatives,  dominated 
by  the  labor  of  annual  operations,  were  driven  by  the  simulation  output  for  number  of 
labor  hours. 

In  spite  of  the  differences  in  confidence  levels,  the  overall  results  should  be  considered 
valid.  The  JTEM  CTM  had  the  median  LCCE,  with  the  lowest  O&S  cost.  This  is 
significant  because  O&S  is  a  recurring  cost,  borne  by  every  C4I  test  organization  that 
implements  one  of  the  alternatives.  Development  costs  of  JTEM  CTM  are  the  largest 
portion  of  its  LCCE,  a  nonrecurring  cost  borne  by  OSD  and  not  borne  by  any  single  C4I 
test  organization. 


Summary  &  Next  Steps 

This  team  was  the  first  to  apply  a  disciplined  systems  engineering  process  to  the  problem 
of  re-engineering  the  business  of  testing  for  C4I  interoperability  certification.  The  JTEM 
project  is  the  only  other  organization  to  examine  this  issue  from  the  perspective  of 
optimizing  a  life-cycle  balanced  solution  to  meet  explicitly  stated  and  quantifiable  needs. 
No  one  has  applied  an  integrated  set  of  computer-based  simulation  tools  to  quantitatively 
predict  the  performance  of  competing  options  and  compare  that  performance  to  life-cycle 
cost.  Knowing  that  C4I  systems  never  perform  in  a  vacuum,  but  always  interoperate  as 
part  of  a  larger  SoS,  developers  and  testers  benefit  from  the  results  of  this  study. 

Ensuring  interoperability  across  services,  and  between  civil  authorities  and  multinational 
organizations,  starts  with  an  effects-based  approach.  Only  by  testing  for  interoperability 
against  performance  measures  linked  to  desired  effects  in  the  battle-space  can  C2  SoS 
support  warfighters  engaged  in  complex  endeavors. 

Based  on  the  insights  into  the  problem  domain  and  potential  solutions,  there  are  areas  of 
further  study.  The  team  believed  the  C4I  acquisition  and  testing  communities  would 
benefit  from  a  dedicated  Joint  C4I  SoS  manager  to  provide  consistency  in  an  evolving 
environment.  Their  role  could  include  documenting  C4I  SoS  capabilities,  long  range  SoS 
capabilities  planning,  testing  requirements  management,  supporting  developmental  and 
operational  testing,  and  addressing  ad  hoc  SoS  configuration  resulting  from  new  threats 
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and  concepts  [Acosta,  et  al].  These  roles  represent  overlap  between  the  acquisition 
community  and  those  responsible  for  communicating  needed  capabilities  to  them.  It  is 
hoped  that  codifying  the  relationship  between  the  Joint  C2  Capability  Portfolio  Manager 
and  the  C2  FCB  will  be  a  move  in  this  direction. 

Next,  as  changes  to  the  SoS  configuration  are  made,  the  likelihood  of  capability  failures 
increases.  The  JC3M  team  believes  risk  management  strategies  should  be  developed  and 
applied  to  the  C4I  SoS.  The  JC3M  team’s  preliminary  list  of  risks  includes  the  lack  of  a 
single  entity  responsible  for  SoS  performance;  lack  of  an  objective,  repeatable,  and 
methodical  approach  to  address  individual  system  problems  impacting  SoS  functionality; 
varying  levels  of  maturity  of  systems  within  the  C4I  SoS  architecture;  and  varied 
interfaces  between  individual  systems. 

And,  finally,  systems  that  are  components  of  the  C4I  SoS  have  their  capabilities  defined 
as  if  they  exist  in  a  vacuum,  and  their  impact  on  C4I  SoS  capabilities  is  generally  not 
considered.  The  DoD  C4I  SoS  acquisition  process  should  require  component  system 
sponsors  to  define  C4I  SoS  level  effects;  establish  a  funding  line  for  SoS  testing;  and 
include  SoS  effectiveness  testing  as  part  of  operational  testing  [Acosta,  et  al]. 
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Alternative  #1 


"System  Capabilities  Review  (SCR)" 


Paper  046 


13 


Alternative  #2 
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Value  Modeling  Overview 
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LCCE  -  Cost  Summary 


Alternatives 

Life-Cycle  Year 

Total  Cost 

($) 

1 

2 

3 

4.. .9 

10 

FEDOS 

1,052,527 

419,497 

419,497 

419,497 

52,200 

5,010,706 

MC3T 

1,169,414 

525,537 

525,537 

525,537 

52,200 

5,975,913 

JTEM-CTM 

1,030,000 

2,470,000 

1,169,414 

558,535 

52,200 

6,972,824 

FCB 

2,323,117 

650,223 

650,223 

650,223 

52,200 

8,127,101 

SCR 

2,121,421 

624,45 1 

624,45 1 

624,451 

52,200 

7,719,232 

Interpretation:  The  delta  between  the  highest  and  lowest  LCCE  ^  $3M, 
which  is  not  a  significant  sum  over  a  ten  year  span. 
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Utility  &  LCCE 


Percentage 

of 

Traceable 

Measures 

Days  to 
Plan 

Evaluation 

Quality 

of 

Planning 

Outputs 

Elasticity 
of  Labor 

Elasticity 

of 

Duration 

Overall 

Utility 

(0-1) 

LCCE 

($  M) 

FEDOS 

0.00 

0.04 

0.39 

0.06 

0.14 

0.63 

5.01 

MC3T 

0.02 

0.05 

0.39 

0.07 

0.17 

0.71 

5.98 

JTEM 

CTM 

0.24 

0.06 

0.40 

0.04 

0.15 

0.89 

6.97 

SCR 

0.24 

0.02 

0.37 

0.05 

0.10 

0.79 

7.72 

FCB 

0.22 

0.05 

0.34 

0.08 

0.18 

0.87 

8.13 
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Utility 


LCCE  vs  Utility 
w 


0.90 

0.80 

0.70 

0.60 

0.50 

0.40 

0.30 

0.20 

0.10 

0.00 


CUM 


$6.97 , 0.89  ✓ 


$8.13,0.87 


)T 

$5.98 , 0.71 


/  $7.72 , 0.79 


IMDXDS  x 


$5.01  ,  0.63 


$- 


$2.00 


$4.00 


$6.00 


$8.00 


$10.00 


Life  Cycle  Cost  Estimate  (SMIL) 


$12.00 


$14.00 
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Way  Ahead:  3  areas 


■a 

o 
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